<html>
<head><meta charset="utf-8"><title>Improving rollups · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html">Improving rollups</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="228592139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228592139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228592139">(Mar 03 2021 at 13:40)</a>:</h4>
<p>As part of the failed MCP for <a href="https://github.com/rust-lang/compiler-team/issues/407">never rolling up changes when touching the <code>compiler/</code> directory</a>, some great conversation on how to improve the rollup process was had in the related <a href="#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/Never.20Rollup.20When.20Changing.20the.20.60compiler.60.E2.80.A6.20compiler-team.23407">Zulip thread</a>. I'd like to continue those conversations here and start experimenting with ways to actually improve the rollup process.</p>
<p>Overall, the goal of rolling up is to get changes into master faster and to make sure the queue does not get backed up. This means we want to maximize the number of PRs in a rollup and simultaneously minimize the number of changes that cause any issues (including failing builds, introducing performance regressions, etc.). Each rollup has an associated risk associated with it. I'd like to start exposing the estimated risk to users of the homu queue, track the success rate of "risky" rollups, and see if we can eventually lower the percentage of rollups that fail. </p>
<p>My first change was to add a "riskiness" label to the rollup when creating it. This at least starts exposing to users of the queue how relatively risky their rollup is predicted to be: <a href="https://github.com/rust-lang/homu/pull/138">https://github.com/rust-lang/homu/pull/138</a></p>



<a name="228610942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228610942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228610942">(Mar 03 2021 at 15:28)</a>:</h4>
<p>it's why in the past we kept rollups to a small number of prs (generally around 5-6) - so that there is a lower risk to it</p>



<a name="228612641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228612641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228612641">(Mar 03 2021 at 15:38)</a>:</h4>
<p>Yes, but that doesn't seem to be followed very often nowadays. Most rollups are around 10 PRs. This labeling should provide more of an indication of when someone is doing something that might lead to a "risky" rollup.</p>



<a name="228614268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228614268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228614268">(Mar 03 2021 at 15:47)</a>:</h4>
<p>yeah i'm aware. the change happened because people felt that prs are remaining "stale" in the queue for a long time and hence we need larger rollups</p>



<a name="228619784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228619784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228619784">(Mar 03 2021 at 16:16)</a>:</h4>
<p>Hopefully if we better quantify risk we can increase the size of Rollups while keeping the risk that they fail low. That’s the hope at least</p>



<a name="228960692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228960692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228960692">(Mar 05 2021 at 14:19)</a>:</h4>
<p>so, my understanding here is that we have two issues we're trying to solve</p>



<a name="228960817"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228960817" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228960817">(Mar 05 2021 at 14:20)</a>:</h4>
<p>the first one is that rollups make the rustc-perf data less granular, which makes it harder for people doing perf triage to figure out what actually made our benchmarks worse</p>



<a name="228960880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228960880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228960880">(Mar 05 2021 at 14:20)</a>:</h4>
<p>and the second one is that we would like to minimize the amount of times rollup fail</p>



<a name="228961003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961003">(Mar 05 2021 at 14:21)</a>:</h4>
<p>That is correct, you can summarize these together as rollups should cause issues as seldom as possible.</p>



<a name="228961051"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961051" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961051">(Mar 05 2021 at 14:21)</a>:</h4>
<p>Ideally all rollups would be merged first try and would never require reverting</p>



<a name="228961156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961156">(Mar 05 2021 at 14:22)</a>:</h4>
<p>personally I don't think the second problem is a big one: if a PR breaks a rollup it will also fail CI when tested on its own, so the number of times CI fails should be the same (if the person doing the rollup figures out the exact PR that failed)</p>



<a name="228961160"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961160" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961160">(Mar 05 2021 at 14:22)</a>:</h4>
<p>Obviously that's a tall order, but I'm convinced we're not as close to that ideal as we could be</p>



<a name="228961191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961191">(Mar 05 2021 at 14:22)</a>:</h4>
<p>but I agree the first problem is a bad one we need to solve</p>



<a name="228961282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961282">(Mar 05 2021 at 14:23)</a>:</h4>
<p>I'm wondering if there is a better solution to it than just having people not put any compiler PR in the rollups</p>



<a name="228961318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961318">(Mar 05 2021 at 14:23)</a>:</h4>
<p>I think your paranethetical is a big assumption, particularly in the case of the large rollups (10+ PRs)</p>



<a name="228961394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961394">(Mar 05 2021 at 14:24)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228961156">said</a>:</p>
<blockquote>
<p>personally I don't think the second problem is a big one: if a PR breaks a rollup it will also fail CI when tested on its own, so the number of times CI fails should be the same (if the person doing the rollup figures out the exact PR that failed)</p>
</blockquote>
<p>I'm not sure I agree with this</p>



<a name="228961582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961582">(Mar 05 2021 at 14:25)</a>:</h4>
<p>Rollups don't always fail due to an actual failed test. Rollups can also fail due to interactions between PRs.</p>



<a name="228961755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961755">(Mar 05 2021 at 14:26)</a>:</h4>
<p>my understanding is that perf doesn't actually need much to run the benchmarks: it just needs cargo + rustc + rustdoc for each commit it wants to benchmark, and it needs that just for x86_64 linux</p>



<a name="228961926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228961926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228961926">(Mar 05 2021 at 14:27)</a>:</h4>
<p>There is still an issue with hanging tests, roughly once a day, so that takes away 4h30m of CI time.</p>



<a name="228962078"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962078" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962078">(Mar 05 2021 at 14:28)</a>:</h4>
<p>I don't recall how long building cargo+rustc+rustdoc takes on CI, but if it's a reasonable amount of time it might make sense to create a small number of "perf builders" on CI that prepare the strict subset of artifacts perf needs for each PR when they detect a rollup</p>



<a name="228962115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962115">(Mar 05 2021 at 14:28)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228962078">said</a>:</p>
<blockquote>
<p>I don't recall how long building cargo+rustc+rustdoc takes on CI, but if it's a reasonable amount of time it might make sense to create a small number of "perf builders" on CI that prepare the strict subset of artifacts perf needs for each PR when they detect a rollup</p>
</blockquote>
<p>you'd still need stage2 artifacts, right?</p>



<a name="228962127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962127">(Mar 05 2021 at 14:29)</a>:</h4>
<p>I guess that's faster than building tools and running tests though</p>



<a name="228962170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962170">(Mar 05 2021 at 14:29)</a>:</h4>
<p>honestly I don't know what perf needs exactly, this is just me thinking aloud :)</p>



<a name="228962219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962219">(Mar 05 2021 at 14:29)</a>:</h4>
<p>Yes, it needs stage 2 artifacts</p>



<a name="228962231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962231">(Mar 05 2021 at 14:29)</a>:</h4>
<p>I'm not sure how this helps though...</p>



<a name="228962362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962362">(Mar 05 2021 at 14:30)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228962231">said</a>:</p>
<blockquote>
<p>I'm not sure how this helps though...</p>
</blockquote>
<p>it means PRs that affect perf can be merged in rollups</p>



<a name="228962372"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962372" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962372">(Mar 05 2021 at 14:30)</a>:</h4>
<p>so we only have problem 2 to worry about</p>



<a name="228962598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962598">(Mar 05 2021 at 14:32)</a>:</h4>
<p>yeah</p>



<a name="228962657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962657">(Mar 05 2021 at 14:32)</a>:</h4>
<p>Ah I see what you're saying: stop perf testing rollups as monoliths and perf test individual changes</p>



<a name="228962671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962671">(Mar 05 2021 at 14:32)</a>:</h4>
<p>yep</p>



<a name="228962795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962795">(Mar 05 2021 at 14:33)</a>:</h4>
<p>This would lead to a lot of perf runs for inconsequential changes - change a markdown file, do a perf run</p>



<a name="228962827"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962827" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962827">(Mar 05 2021 at 14:33)</a>:</h4>
<p>there could be a new flag to bors, like <code>@bors r+ rollup perf</code></p>



<a name="228962855"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962855" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962855">(Mar 05 2021 at 14:33)</a>:</h4>
<p>which tells it to measure perf separate from the rollup</p>



<a name="228962950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228962950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228962950">(Mar 05 2021 at 14:34)</a>:</h4>
<p>or just add heuristics to only produce those artifacts for PRs changing <code>compiler/</code></p>



<a name="228963022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228963022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228963022">(Mar 05 2021 at 14:34)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228962950">said</a>:</p>
<blockquote>
<p>or just add heuristics to only produce those artifacts for PRs changing <code>compiler/</code></p>
</blockquote>
<p>or bootstrap or the standard library or ... I still feel like there should be a human involved</p>



<a name="228963169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228963169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228963169">(Mar 05 2021 at 14:35)</a>:</h4>
<p>This is why I'd like to start collecting more statistics on rollups. It'd be nice to talk numbers here. For instance, how many rolled up PRs labeled rollup=always lead to issues</p>



<a name="228963565"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228963565" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228963565">(Mar 05 2021 at 14:38)</a>:</h4>
<blockquote>
<p>For instance, how many rolled up PRs labeled rollup=always lead to issues</p>
</blockquote>
<p>I feel like this is subjective too though, what are "issues"? failed rollups? perf regressions? perf improvements? what's the cutoff for a perf change?</p>



<a name="228963654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228963654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228963654">(Mar 05 2021 at 14:38)</a>:</h4>
<p>Failed roll ups are a pretty obvious starting point albeit unrelated to the current topic of performance</p>



<a name="228964193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964193">(Mar 05 2021 at 14:42)</a>:</h4>
<p>sure, seems fine to collect those statistics, but I don't think it will help much with deciding what to do about rollups that affect perf</p>



<a name="228964229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964229">(Mar 05 2021 at 14:42)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228963169">said</a>:</p>
<blockquote>
<p>This is why I'd like to start collecting more statistics on rollups. It'd be nice to talk numbers here. For instance, how many rolled up PRs labeled rollup=always lead to issues</p>
</blockquote>
<p>what's your plan for collecting those stats?</p>



<a name="228964557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964557">(Mar 05 2021 at 14:44)</a>:</h4>
<p>Not sure yet. We just need to know track somewhere created rollups. I believe that’s not stored anywhere but GitHub</p>



<a name="228964645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964645">(Mar 05 2021 at 14:45)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228964557">said</a>:</p>
<blockquote>
<p>Not sure yet. We just need to know track somewhere created rollups. I believe that’s not stored anywhere but GitHub</p>
</blockquote>
<p>you could look for merge commits that say "Rollup of X pull requests"</p>



<a name="228964754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964754">(Mar 05 2021 at 14:46)</a>:</h4>
<p>e.g. here's a rollup merge and a non-rollup merge:</p>
<div class="codehilite"><pre><span></span><code>$ git log --author=bors
commit ec7f258d543e1ac7d0b94435972331e85da8c509
Merge: 7f32f62aa5c 06630f7b5f6
Author: bors &lt;bors@rust-lang.org&gt;
Date:   Thu Mar 4 11:02:13 2021 +0000

    Auto merge of #82756 - JohnTitor:rollup-e4ij7h6, r=JohnTitor

    Rollup of 8 pull requests

    Successful merges:

     - #80527 (Make rustdoc lints a tool lint instead of built-in)
     - #82310 (Load rustdoc&#39;s JS search index on-demand.)
     - #82315 (Improve page load performance in rustdoc)
     - #82564 (Revert `Vec::spare_capacity_mut` impl to prevent pointers invalidation)
     - #82697 (Fix stabilization version of move_ref_pattern)
     - #82717 (Account for macros when suggesting adding lifetime)
     - #82740 (Fix commit detected when using `download-rustc`)
     - #82744 (Pass `CrateNum` by value instead of by reference)

    Failed merges:

    r? `@ghost`
    `@rustbot` modify labels: rollup

commit 7f32f62aa5ceba1b795f3702e502d8473238be6b
Merge: 6f7673d077a 61114453ae5
Author: bors &lt;bors@rust-lang.org&gt;
Date:   Thu Mar 4 05:46:43 2021 +0000

    Auto merge of #82304 - LeSeulArtichaut:unpretty-ast, r=spastorino

    Add `-Z unpretty` flags for the AST

    Implements rust-lang/compiler-team#408.
    Builds on #82269, but if that PR is rejected or stalls out, I can implement this without #82269.
    cc rust-lang/rustc-dev-guide#1062
</code></pre></div>



<a name="228964960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228964960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228964960">(Mar 05 2021 at 14:47)</a>:</h4>
<p>so, I looked a bit at the CI logs, and a <code>./x.py build --stage 2 library/std</code> on CI takes 0:16:25</p>



<a name="228965029"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228965029" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228965029">(Mar 05 2021 at 14:48)</a>:</h4>
<p>Yea I might just make a script that calls the GitHub api, but the missing info is what roll up status that each PR received</p>



<a name="228965202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228965202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228965202">(Mar 05 2021 at 14:49)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228965029">said</a>:</p>
<blockquote>
<p>Yea I might just make a script that calls the GitHub api, but the missing jnfo is what roll up status the each PR received</p>
</blockquote>
<p>I think we could have a <code>rollup_stats</code> table on homu's database that records each PR included in each rollup and their rollup state</p>



<a name="228965738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228965738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228965738">(Mar 05 2021 at 14:52)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228964960">said</a>:</p>
<blockquote>
<p>so, I looked a bit at the CI logs, and a <code>./x.py build --stage 2 library/std</code> on CI takes 0:16:25</p>
</blockquote>
<p>this also includes building bootstrap and downloading from sccache and installing llvm, so if we also build cargo and rustdoc I think we should easily be between 20 and 25 minutes for every commit we generate artifacts in CI for</p>



<a name="228965858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228965858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228965858">(Mar 05 2021 at 14:53)</a>:</h4>
<p>so with really handwavy math we should be able to generate artifacts for every PR included in a &lt;= 10 rollup with one or two builders</p>



<a name="228965866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228965866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228965866">(Mar 05 2021 at 14:53)</a>:</h4>
<p>[in a call for the next 2 hours]</p>



<a name="228986057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228986057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228986057">(Mar 05 2021 at 16:53)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228963169">said</a>:</p>
<blockquote>
<p>This is why I'd like to start collecting more statistics on rollups. It'd be nice to talk numbers here. For instance, how many rolled up PRs labeled rollup=always lead to issues</p>
</blockquote>
<p>don't have numbers but generally very few rollup=always prs fail. Largely because - most reviewers/other lurkers often use rollup=always sparingly and mostly for minor prs. </p>
<p>Though getting stats for rollup=maybe and rollup=iffy would be nice and we can check probably how many of those should have been rollup=never to start with</p>



<a name="228991210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/228991210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#228991210">(Mar 05 2021 at 17:23)</a>:</h4>
<p>out of curiosity, are there stats on how long a normal priority pr (no rollup, no p=1, no rollup=never etc) remains in the queue after being approved and how often it has to be rebased on average?</p>



<a name="229020386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229020386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229020386">(Mar 05 2021 at 20:35)</a>:</h4>
<p>I don’t believe we have that information either. It would nice to know how long items are in the queue and to see how various factors influence that</p>



<a name="229049707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229049707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229049707">(Mar 06 2021 at 00:57)</a>:</h4>
<p><span class="user-mention silent" data-user-id="217864">matthiaskrgr</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/228991210">said</a>:</p>
<blockquote>
<p>out of curiosity, are there stats on how long a normal priority pr (no rollup, no p=1, no rollup=never etc) remains in the queue after being approved and how often it has to be rebased on average?</p>
</blockquote>
<p>it varies a lot. it depends on - </p>
<ul>
<li>how long the queue is, especially if we are doing smaller rollups it means PRs will last longer </li>
<li>number of rollup=never prs, since they are rolled up individually and get priority over non-never prs, other prs often have to wait for them to be merged which is why sometimes i'm fine to allow rollups to jump never-prs </li>
<li>
<p>stuff like failed rollups or tree being closed due to other reasons can sometimes  mean a few prs have to wait longer </p>
</li>
<li>
<p>another pattern I used to observe often but not much off late is the "vortex pattern" of a pr remaining in the middle of the queue because new ones get added on the top and then included in a rollup, but before the ones in the middle get rolled up more prs get added and the cycle continues. (I guess having larger rollups reduced the chances of it occurring) </p>
</li>
</ul>
<p>overall i'd think a PR remains for 1-2 days in normal cases, and 3-4 days or a week in worse cases but these are just speculations</p>



<a name="229089552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229089552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229089552">(Mar 06 2021 at 11:15)</a>:</h4>
<p>so the "vortex pattern"  applies to to every pr that is neither rolled up nor prioritized in other way..?</p>



<a name="229089774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229089774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229089774">(Mar 06 2021 at 11:19)</a>:</h4>
<p>I wonder if it would be useful to add like p=0.4 to a pr once it has been approved but a merge conflict occurs.<br>
That means that once you resolve the merge conflict, your pr automatically gets prioritized over prs that don't need rebase.<br>
PRs that get conflicts over and over again would naturally gain more cumulative priority and would then merge faster and small prs that are "stable" remain at the bottom and can be picked up by a rollup.</p>



<a name="229098180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229098180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229098180">(Mar 06 2021 at 13:31)</a>:</h4>
<p>Well, we already prioritize older PRs, though it's not quite the same.</p>



<a name="229115244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Improving%20rollups/near/229115244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Improving.20rollups.html#229115244">(Mar 06 2021 at 17:42)</a>:</h4>
<p><span class="user-mention silent" data-user-id="217864">matthiaskrgr</span> <a href="#narrow/stream/242791-t-infra/topic/Improving.20rollups/near/229089552">said</a>:</p>
<blockquote>
<p>so the "vortex pattern"  applies to to every pr that is neither rolled up nor prioritized in other way..?</p>
</blockquote>
<p>not all the PRs but a few of those used to.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>